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ABSTRACT 

Evaluations of Chapter 1 programs usually center on 
one of three questions: (1) Does the program constitute an 
appropriate Federal investment? (2) Does the program have the 
potential to benefit students? and (3) Is the program feasible enough 
to implement? Often, evaluators and legislators disagree over which 
of these questions is most important. In the case of Chapter 1, 
getting the money to the right children is an outcome of great 
interest, and many congressmen assume that once that is accomplished, 
it is up to the district to see that the funds are used effectively. 
At the moment, the most asked question concerns whether and how well 
a proposed program can be implemented. This is evident in the 
National Institute of Education's (NIE) mandate to study the 
reauthorization of Chapter 1: of the seven items of information 
requested by Congress, six concerned implementation, and only one 
concerned effectiveness. It is likely that the 1987 reauthorization 
hearings will reflect this configuration of questions. Thus, the NIE 
study will focus on four implementation questions (administration, 
targeting, program design, and services). This study differs from 
others in its efforts to review existing evidence (rather than 
gathering new data) and in the design of its implementation study, 
which emphasizes the way context can influence the fate of the 
Chapter 1 program. Two very different kinds of contexts can have an 
impact: (1) local organization factors and the management and 
budgetary constraints faced by Chapter 1 directors, and (2) the 
educational "mood" of the district, and nation, regarding the desired 
direction of education. (KH) 
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The National Assessment of the Chapter 1 Program 

Mary M. Kennedy 
National Institute of Education 

After working in the Chapter 1 program and discovering its long 
history of evaluation, I have become increasingly aware that there are 
certain basic questions that are asked on almost any evaluation occasion. 
And although I became aware of these questions within the context of the 
Chapter 1 program, I think the questions are probably universally applicable. 
That is, they are likely to be asked by almost any legislator or educational 
administrator who must decide whether or not to authorize e particular 
program. The trick for the evaluator is not to discover what these questions 
are, but rather to discover what form they are taking at the moment. The 
first of these questions has to do with whether the program under 
consideration constitutes a legitimate or appropriate investment in light of 
the funding agency's mission. In federal education policy, this question 
plays itself out as one of defining the appropriate federal role in 
education. But the question of appropriateness comes up in debates about 
state and local policy as well. For instance, debates about the relative 
emphasis on academic versus vocational education programs in secondary 
schools are often couched in terms of appropriateness for the agency. The 



second question is whether the program has the potential to benefit 
students — that is, whether it is or could be effective. Anyone who must make 
a funding decision will want some assurances regarding the benefits that are 
likely to occur as a result of the program. Finally, there is the question 
of whether the program is feasible, or could be made feasible, to implement. 

While I have argued that these questions are universally applicable, 
I have also argued that they are not universally applied, or at least not 
uniformly applied. They receive different emphases by different people at 
different times and in different circumstances. Furthermore, each question, 
regardless of its form, poses special problems for the evaluator. Let me 
illustrate these points by first describing the current form these questions 
are taking in the Chapter 1 program, and then telling you how we are 
addressing them. 

Questions regarding the appropriateness of federal investments in 
this area were big questions when the program was first legislated in 1965. 
Tangled up in the decision about whether to authorize were questions about 
whether or not private schools should receive money, and whether the Federal 
government should give money to segregated schools. These are special forms 
of the question of appropriateness of the funding agency's — in this case, the 
federal government's — involvement. Once the initial decision was made, these 
questions tended to recede for the next several years. They have recently 
started to come back to the forefront as part of a larger debate about the 
appropriateness of the federal government's involvement in a number of areas. 
Even now, though, these questions are raised more by conservative interest 
groups outside of government, and occasionally by individuals within the 



Education Department, than by members of the relevant Congressional 
committees. The Congressional staff with whom I spoke when developing the 
plan for this study continue to assume that this program constitutes a 
legitimate investment. Of course, questions regarding legitimacy are not 
empirical questions anyway, and I would not recommend that evaluators address 
them. But because these questions are value-laden, they sensitize all of the 
participants. To that extent, evaluators need to be aware that such 
questions are being asked, for they can sensitize evaluation findings as 
well. 

Questions regarding effectiveness also present special problems for 
evaluators. They are the primary questions that evaluators have been trained 
to answer, they are the questions evaluators tend to think are most 
important, and they are the questions evaluators want most to answer. 
But they often are not the foremost question on the minds of those we tend to 
call decision makers. There are several reasons for this. One is that 
decision makers tend to think of outcomes differently than evaluators do. In 
the case of Chaj *:er 1, for instance, getting the money to the right children 
is an outcome of great interest, and many Congressmen tend to assume that 
once that is accomplished, it is up to the district to see that the funds 
have an effect on those children. For these legislators, evidence that the 
funds are flowing in the right direction is evidence of program effective- 
ness. Another reason they place less emphasis on formal evidence of effect- 
iveness is that they tend to accept other kinds of evidence regarding 
effectiveness. The legislative system, like the legal system, is designed to 
accept primarily verbal testimony. And verbal testimony tends to emphasize 



anecdotes and first-hand experiences. In the case of the Chapter 1 program, 
there is yet a third reason why our audience has not emphasized 
effectiveness, and that has to do with the age of the program. It has 
existed for 20 years now, and has been evaluated on a number of occasions. 
One of the things I was impressed by when I interviewed Congressional staff 
was their familiarity with these evaluation findings. This is not to say 
that they had all reached the same conclusions about the program, however. 
Their perceptions varied from enthusiastic regarding its successes to 
disappointed by it. Yet because they all had a broad range of evidence to 
draw on, questions regarding the impact of the program on children tended to 
be asked only as an afterthought. 

The third universally-applicable question, having to do with whether 
or how well the program can be implemented, is receiving the largest emphasis 
in the Chapter 1 program right now. This emphasis is evident in the mandate 
for our study: of the seven items in the mandate for this study, only one 
asked about effectiveness. The other six asked for the kind of descriptive 
information frequently used to indicate how, or how well, a program is being 
implemented. The bill requires information about: 

0 The recipients, 

0 The services, 

0 The background and training of teachers and staff, 

0 Coordination of services, 

0 How funds are allocated among schools, 

0 Effectiveness, and 

0 How certain key provisions are implemented. 
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Notice that most of these items refer only descriptive data, and do 
not reveal the questions that lie behind the requirements. Consultations with 
congressional staff enabled us to learn how the required descriptive data 
were expected to be useful — that is, what the Congress expected them to 
indicate about the program. In addition, these consultations exposed us to a 
variety of other issues not even hinted at in the mandate. The questions 
raised during my interviews with Hill staff included questions about the 
implications of the most recent legislative changes, about what the states 
are doing now, about the role of parents, about services to special 
populations such as children with limited English proficiency, about whether 
the rules are accomplishing what that were intended to accomplish, whether 
they are unnecessarily burdensome, and so forth. Like the writte.i mandate, 
they emphasized implementation more than effectiveness. 

This brief review gives you some idea of the configuration of 
questions that probably will be asked during the 1987 reauthorization 
hearings. Questions regarding appropriateness of the federal role will 
probably be asked. While these are not empirical questions, their very 
existence sensitizes participants, and also makes evaluation data more 
sensitive. Questions about effectiveness are being asked largely as an 
afterthought, even though there are disparities in perceptions of the 
program's effectiveness. Questions regarding implementation predominated, 
both in the written mandate and in conversations. The dominance of this area 
over others is partly due to the fact that the rules for the program were 
recently altered, and most legislative staff wanted to know whether these 
clianges in law helped or hindered districts as they attempted to provide 
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services to disadvantaged youth. 

As a result of these conversations, we have decided that the vast 
majority of our procured studies should concentrate on implementation 
questions. Within this broad field, we have identified four major 
substantive areas which these procurements will focus on. One of these is 
Administration . Falling within this area are a variety of questions 
having to do with the adequacy of various requirements, and the 
burdensomeness of others. This area also includes questions regarding the 
state role, and interactions among federal, state and local agencies. The 
second area is Taraeting , in which fall questions about the selection of 
both the schools and the students which will participate in this program. 
This area appears to be continually of interest to the Congress, in part 
because the program is intended to serve a particular kind of child, and in 
part because there is no obvious or natural method for identifying these 
children, so that the possibility for slippage between intend and reality is 
great. The third area is Program Design » which includes questions not 
only about the design of local programs, but also about the processes used to 
develop these designs — how parents are involved, for instance, and how 
evaluation results are used. Finally, we have the area of Services , 
which includes questions about the nature, quantity, and quality of services 
provided to students. 

Anyone who is familiar with evaluations of Chapter 1, or with 
evaluations in general, will probably notice that the areas we are studying 
are indeed similar to areas studied by others. These similarities reflect, I 
think, the universality of the questions we have been asked to address. In 
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fact, because many of our research questions are similar to questions asked 
by earlier evaluators of this program, many of our studies are being designed 
to replicate earlier studies. We hope, by doing this, to be able to make 
cross-time comparisons regarding how the program is being carried out. 

Now let me describe two ways in which I think our study differs 
from others. One has to do with how we will handle the effectiveness 
question, the other with how we have designed our studies in the 
implementation areas, 

V/ith regard to effectiveness, it seemed to us unnecessary to conduct 
yet another large, expensive study of the effectiveness of the Chapter 1 
program, which could then be criticized like those before it for its design 
and its choice of measures. Yet it did seem that there were some legitimate 
questions that could be addressed. For instance, there was considerable 
disparity among perceptions of the program's effectiveness, and we felt we 
could help Congress by reviewing the wealth of information that already 
existed about the program, summarizing it for them and explaining why there 
were discrepancies among study findings. In addition, several people did ask 
about the relationship between certain requirements and effectiveness. For 
instance, there was a belief that many districts implement pullout programs 
largely to comply with fiscal accounting rules, not because these are 
necessarily effective program designs. There seemed to be an interest in 
knowing more about how the Congress, through its rule-making, could 
facilitate local program design so that the effectiveness of the program 
could be enhanced. We therefore decided to provide the Congress with a 
separate report which focuses solely on questions of effectiveness, but which 



reviews existing evidence rather than gathering new evidence. We hope this 
report will explain some of the methodological difficulties associated with 
trying to measure effectiveness, as well as summarize the relationships 
between program characteristics and student outcomes that have been 
identified. 

The second way in which our study differs from others is in the way 
in which our studies of implementation are being designed. We have the 
advantage of 20 years of experience in studying this program, and have tried 
to benefit from that experience. The most important methodological 
assumption we make is that the program exists within a context. Stated so 
simply, it hardly seems astounding, but it is the single most important 
lesson we derived from the literature. It means that when we look to see how 
districts are implementing certain provisions we must also look to see what 
aspects of their particular situation facilitate or inhibit their ability (or 
desire) to implement th^2se provisions. Here are some examples of how we have 
taken context into account in designing some of our studies, 

1. We wanted a study that would tell us how districts identify 
schools as eligible to receive a Chapter 1 program, and how they select the 
actual schools and students that will participate in the program. We also 
wanted a study that vould tell us how they allocate their resources among 
participating schools. While it would be possible to address either of these 
questions without regard for what else is happening in school districts, we 
have instead designed these two studies to include data regarding students 
who have been placed in other categorical programs such as bilingual 
education and special education, and data regarding how resources from 
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these other programs are distributed among schools. 

2. We wanted a study that would give us good descriptive information 
about the services that Chapter 1 students receive. In fact, we wanted two 
studies, one that would provide survey statistics about services, and another 
that would provide more intensive, qualitative descriptions of services. 
Because of our awareness of context, we have designed both of these studies 
to include data regarding services outside the immediate Chapter 1 classroom. 
Survey questionnaires, for instance, will go to school principals. Chapter 1 
teachers, regular teachers, special education teachers, and others, if we 
feel it would be appropriate. For the qualitative study, we won't merely 
observe Chapter 1 classrooms, but will actually follow students through their 
entire school day, to see how this one segment of their education fits into 
the whole pattern. 

In thinking about how to come to terms with context, we now realize 
that there are two very different kinds of contexts that can influence the 
fate of the Chapter 1 program. One of them includes local organizational 
factors such as the distribution of eligible children in the district, the 
arrangement of classrooms and teachers among schools, and the variety of 
management or budgetary constraints that Chapter 1 directors face as they 
design and implement their programs. The other is more subtle; xt is the 
mood of the district, and even the mood of the nation, regarding what is 
important in education and where education should be going. The Chapter 1 
program exists not only in an organizational context, but also in a cultural 
climate which influences its form and purpose. I don't know if we would have 
realized that there was a cultural climate to take into consideration if the 
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nation were not now in the midst of a major reform movement in education. 
But given that reform movement, we have also become concerned about how it 
might bear on the Chapter 1 program. The issues here are far more subtle and 
complex than the organizational issues I just alluded to. Beatrice Birman 
will now discuss these issues. 
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